Object class identification, verification of object image synthesis

ABSTRACT

A method of identifying an object class uses a model based upon appearance parameters derived by comparing images of objects of different classes. The model includes a representation of a probability density function describing a range over which appearance parameters may vary for a given class of object, the model further including a defined relationship between the appearance parameters and the probability density function. The method generates appearance parameters representative of an unknown object, estimates an appropriate probability density function for the unknown object using the defined relationship between the appearance parameters and the probability density function, then iteratively modifies at least some of the appearance parameters within limits determined using the probability density function to identify the object class.

BACKGROUND

1. Technical Field

The present invention relates to the identification or verification ofan object class, and relates also to the synthesis of images of objects.The invention relates particularly though not exclusively to theidentification or verification of faces, and relates also to thesynthesis of images of faces.

2. Related Art

Many known methods of face identification utilise a universal face spacemodel indicative of facial features of a non-homogeneous population.Commonly, the universal face space model is represented as a set ofappearance parameters that are best able to represent variations betweenfaces in a restricted dimensional space (see For example U.S. Pat. No.5,164,992; M. A. Turk and A. P. Pentland). A face to be identified isconverted into a set of appearance parameters and then compared withsets of appearance parameters indicative of known faces.

Recently published face identification methods include the activeappearance method and active shape method (G. J. Edwards, C. J. Taylor,and T. F. Cootes. Face recognition using Active Appearance Models. In5^(th) European Conference on Computer Vision, pages 581–595, 1998; T.F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Active ShapeModels—their training and application. Computer Vision and ImageUnderstanding, 61(1):38–59, January 1995). The active appearance methodcomprises a universal face space model with which an unknown face iscompared, and further includes pre-learned knowledge indicating how toadjust appearance parameters in universal face space in order to match aface synthesised using the model to an unknown face. Using thepre-learned knowledge is advantageous because it allows the requirednumber adjustment iterations to be minimised.

If the facial appearance of each individual was unchanging, and everyimage of each individual was identical, then each individual could berepresented by a single point in universal face space. However, thefacial appearance of an individual may vary in response to a number offactors, for example changes of expression, pose or illumination. Thevariability of appearance parameters representative of the appearance ofan individual under changes of expression, pose, illumination or otherfactors can be expressed as a probability density function. Theprobability density function defines a volume in universal face spacewhich is considered to correspond to a given individual. So for example,a series of images of an individual, with a variety of expressionsshould all fall within the volume described by the probability densityfunction in universal face space.

In known face identification methods a single probability densityfunction is generated which is applied to all individuals. This is donecentering the probability density function on a mean parameter vectorfor a given individual.

BRIEF SUMMARY

It is an object of the first aspect of the invention to provide animproved object class identification or verification method.

According to a first aspect of the invention there is provided a methodof identifying an object class using a model based upon appearanceparameters derived by comparing images of objects of different classes,the model including a representation of a probability density functiondescribing a range over which appearance parameters may vary for a givenclass of object, the model further including a defined relationshipbetween the appearance parameters and the probability density function,wherein the method comprises generating appearance parametersrepresentative of an unknown object, estimating an appropriateprobability density function for the unknown object using the definedrelationship between the appearance parameters and the probabilitydensity function, then iteratively modifying at least some of theappearance parameters within limits determined using the probabilitydensity function to identify the object class.

The method according to the first aspect of the invention providesenhanced identification of the class of an unknown object.

Preferably, a threshold level of the probability density function isdetermined, and the appearance parameters are constrained to have aprobability density greater than the threshold.

Suitably, the objects are faces, and a given class of object is a facehaving a particular identity. The objects may alternatively be hands,livestock, cars, etc. In each case, the class of object is a specificexample of that object, for example a particular person's hand, aparticular horse, or a particular model of car.

The first aspect of the invention is advantageous because it uses thefact that different individual's faces will vary in different ways, tolimit the range over which appearance parameters may vary whenidentifying a face.

Suitably, the relationship between the appearance parameters and theprobability density function is a relationship between averageappearance parameters determined for each class of object, andprobability density functions associated with each class of object.

The probability density function may be any suitable function, forexample a Gaussian function.

Suitably, the probability density function is approximated as a Gaussianwith a given covariance matrix.

Suitably, the model is the Active Appearance Model. The model mayalternatively be the Active Shape Model.

The first aspect of the invention may also be used for example toimprove tracking of individuals by taking into account the predictedvariability in the appearance of a given individual. By providingstronger constraints on the expected variation in appearance of anindividual, matching the model to an image or a sequence can be mademore robust.

According to a second aspect of the invention there is provided a methodof verifying the identity of an object class using a model based uponappearance parameters derived by comparing images of objects ofdifferent classes, the model including a representation of a probabilitydensity function describing the variation of appearance parameters for agiven class of object, the model further including a definedrelationship between the probability density function and the appearanceparameters, wherein for an object class for which verification willsubsequently be required, the method comprises using a series of imagesof that object class to generate a representation of a specificprobability density function describing the variation of appearanceparameters for that object class and, during subsequent verification ofthe object class, comparing an image of an object of an unknown classwith the object of known class by generating appearance parametersrepresentative of the object of unknown class and iteratively modifyingat least some of the appearance parameters within limits determinedusing the specific probability density function.

The term ‘verification’ is intended to mean that the class of an unknownobject is checked to see whether it corresponds with a particular objectclass.

Preferably, a threshold level of the probability density function isdetermined, and the appearance parameters are constrained to have aprobability density greater than the threshold.

Suitably, the objects are faces, and a given class of object is a facehaving a particular identity. The objects may alternatively be hands,livestock, cars, etc. In each case, the class of object is a specificexample of that object, for example a particular person's hand, aparticular horse, or a particular model of car.

The second aspect of the invention is advantageous because it uses thefact that different individual's faces will vary in different ways, tolimit the range over which appearance parameters may vary when verifyingthe identity of a face.

The method according to the second aspect of the invention may provideimproved verification of the class of an unknown object. For example theface of a given individual may have appearance parameters that all fallwithin a relatively compact probability density function. The modelaccording to the second aspect of the invention will only verify that animage is an image of that individual's face if its appearance parametersfall within that compact probability density function. A prior artverification method using a single global probability density function,which will have a greater volume, may erroneously verify the identity ofan image of an individual, if the appearance parameters fall within theglobal probability density function (the verification will be erroneousif the appearance parameters would have fallen outside of the relativelycompact probability density function that would have been provided bythe second aspect of the invention).

Suitably, the relationship between the appearance parameters and theprobability density function is a relationship between averageappearance parameters determined for each class of object, andprobability density functions associated with each class of object.

The probability density function may be any suitable function, forexample a Gaussian function.

Suitably, the probability density function is approximated as a Gaussianwith a given covariance matrix.

Suitably, the model is the Active Appearance Model. The model mayalternatively be the Active Shape Model.

According to a third aspect of the invention there is provided a methodof generating a synthesised image of an object class using a model basedupon appearance parameters derived by comparing images of objects ofdifferent classes, the model including a representation of a probabilitydensity function describing the variation of appearance parameters for agiven class of object, the model further including a definedrelationship between the probability density function and the appearanceparameters, wherein for an object class to be synthesised, the methodcomprises using a series of images of that object class to generate arepresentation of a specific probability density function describing thevariation of appearance parameters for that object class, wherein thesynthesised image of the object class is generated using appearanceparameters confined within limits determined using the specificprobability density function.

The third aspect of the invention is advantageous because it providesobject class specific limits within which appearance parameters of thesynthesised image should lie.

Preferably, a threshold level of the probability density function isdetermined, and the appearance parameters are constrained to have aprobability density greater than the threshold.

Suitably, the objects are faces, and a given class of object is a facehaving a particular identity. The objects may alternatively be hands,livestock, cars, etc. In each case, the class of object is a specificexample of that object, for example a particular person's hand, aparticular horse, or a particular model of car.

The third aspect of the invention is advantageous because it uses thefact that different individual's faces will vary in different ways, tolimit the range over which appearance parameters may vary whensynthesising an image of a face.

Suitably, the relationship between the appearance parameters and theprobability density function is a relationship between averageappearance parameters determined for each class of object, andprobability density functions associated with each class of object.

The probability density function may be any suitable function, forexample a Gaussian function.

Suitably, the probability density function is approximated as a Gaussianwith a given covariance matrix.

Suitably, the model is the Active Appearance Model. The model mayalternatively be the Active Shape Model.

In general, a large number of appearance parameters are required torepresent faces with sufficient detail to allow good face recognition.An alternative way of saying this is that a universal face space modelmust have a large number of dimensions in order to represent faces withsufficient detail to allow good face recognition.

Typically, around 100 appearance parameters are required to representfaces in universal face space model with sufficient detail to allowaccurate identification of faces. However, the number of appearanceparameters that vary significantly for a given individual face is muchless than 100 (typically it is 30), and the remaining appearanceparameters (typically 70) are substantially redundant. Existing faceidentification models attempt to identify a face by varying all of theappearance parameters that make up the universal face space model.Similarly, all available appearance parameters are varied whenattempting to identify the class of an object other than a face.

It is an object of the fourth aspect of the invention to provide animproved object class verification method.

According to a fourth aspect of the invention there is provided a methodof verifying the identity of a class of an object using a model basedupon appearance parameters derived by comparing images of objects ofdifferent classes, the method comprising comparing an object of anunknown class with an object of a known class which has beenincorporated into the model, wherein the appearance parameters aretransformed into a set of transformed parameters defined in a newco-ordinate space, the new co-ordinate space being chosen such that aplurality of the transformed parameters do not vary in that co-ordinatespace, the transformed parameters being compared with a set ofparameters representing the object of known class in that co-ordinatespace.

The inventors have realised that a lesser number of degrees of freedomare required when verifying whether an image is of an object of a givenclass (for example, verifying the identity of an individual using animage of that individual's face), than are required to recognise anobject class in an image containing an object of an unknown class (forexample, attempting to recognise the identity of a face contained in animage). Furthermore, by rotating the co-ordinate space, the number ofparameters that must be varied in order to verify the identity of aclass of an object may be reduced significantly.

Suitably, the objects are faces, and a given class of object is a facehaving a particular identity. The objects may alternatively be hands,livestock, cars, etc. In each case, the class of object is a specificexample of the object, for example a particular person's hand, aparticular horse, or a particular model of car.

Suitably, the model is the Active Appearance Model. The model mayalternatively be the Active Shape Model.

When the method according to the fourth aspect of the invention is used,the number of parameters required for verification is typically reducedfrom approximately 100 to approximately 30. The fourth aspect of theinvention thus makes it possible to perform image matching using a model‘tuned’ to a specific individual (i.e. using a new co-ordinate spacewhich minimises the number of transformed parameters required forverification of that individual). This provides faster and more reliableverification.

According to a fifth aspect of the invention there is provided a methodof transmitting a series of images of an object of a given class using amodel based upon appearance parameters derived by comparing images ofobjects of different classes, wherein appearance parameters aretransformed into a set of transformed parameters defined in a newco-ordinate space, the new co-ordinate space being chosen such that aplurality of the transformed parameters do not vary in that co-ordinatespace, transformed parameters which do not vary are transmitted to areceiver together with a description of the transformation, andtransformed parameters which do vary are subsequently transmitted to thereceiver for each image of the series of images, an inversetransformation being used to transform the received transformedparameters back to appearance parameters at the receiver.

Preferably, the appearance parameters are used to generate a synthesisedimage at the receiver.

According to a sixth aspect of the invention there is provided a methodof storing a series of images of an object of a given class using amodel based upon appearance parameters derived by comparing images ofobjects of different classes, wherein appearance parameters aretransformed into a set of transformed parameters defined in a newco-ordinate space, the new co-ordinate space being chosen such that aplurality of the transformed parameters do not vary in that co-ordinatespace, transformed parameters which do not vary are stored at a storagemedium together with a description of the transformation, andtransformed parameters which do vary are also stored at the storagemedium for each image of the series of images.

Suitably, the transformation is a linear transformation.

Alternatively, the transformation is a non-linear transformation.

Preferably, the objects are faces, and a given class of object is a facehaving a particular identity.

Preferably, the model is the Active Appearance Model.

The Active Appearance Model is a known face identification method (G. J.Edwards, C. J. Taylor, and T. F. Cootes. Face recognition using ActiveAppearance Models. In 5^(th) European Conference on Computer Vision,pages 581–595, 1998). During training of the Active Appearance Model,differences between a synthetic image and a target image are monitored,and a regression matrix relating displacement of the synthetic image(generated by appearance parameters) to the measured differences betweenthe synthetic image and target image is determined. During objectidentification using the Active Appearance Model, the regression matrixis used to drive the model to an object class identification which givesa minimal error.

It is an object of the seventh aspect of the invention to provide animproved object class identification method.

According to a seventh aspect of the invention there is provided amethod of identifying an object class using a model based on appearanceparameters derived by comparing images of objects of different classes,the model including a pre-learned estimated relationship describing theeffect of perturbations of the appearance parameters upon an imagedifference, the image difference comprising a set of elements describingthe difference between an image of an object generated according to themodel and an image of the object itself, wherein a specific relationshipis pre-learned for a specific class of object, and a different specificrelationship is pre-learned for a different class of object.

Suitably, the relationship is defined as a regression matrix.

Suitably, the objects are faces, and a given class of object is a facehaving a particular identity. The objects may alternatively be hands,livestock, cars, etc. In each case, the class of object is a specificexample of the object, for example a particular person's hand, aparticular horse, or a particular model of car.

The seventh aspect of the invention is advantageous because it providesa regression matrix which is specific to an object class. In the case offace recognition, the regression matrix is specific to a face having aparticular identity, and this provides a more robust and accuratesearch.

Each aspect of the invention, as described above is useful in isolation.However, the aspects of the invention can be combined together toprovide faster and more robust object class identification and/orverification.

The first aspect of the invention can be used to provide an object classspecific probability density function, which may then be used by themethod according to the fourth aspect of the invention or the methodaccording to the seventh aspect of the invention.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

A specific embodiment of the first and second aspects of the inventionwill now be described by way of example only.

The invention may be applied to a model based upon a universal facespace model indicative of facial features of a non-homogeneouspopulation.

The variability of appearance parameters for a face of a given identity,under changes in expression, pose, illumination etc, can be expressed inuniversal face space model as a probability density function (PDF). Itis assumed that, for a PDF centred on the average appearance values of agiven face that any expression, pose or illumination of that face shouldbe described by appearance parameters that lie within the PDF.

In prior art face recognition methods, a single PDF is used to describethe variation of all faces irrespective of identity. However, inpractise, different faces will vary in different ways. The first aspectof the invention takes account of this by defining a relationshipbetween the appearance parameters representative of differentindividual's faces and their PDF's. This allows a PDF specific to aparticular face to be predicted for an unknown face on the basis of asingle image of that face.

The model includes PDF's computed for certain faces for which there aremany examples, including variations in pose, illumination, expression,etc. These ‘well-known’ faces are at various locations in the universalface space (as defined by appearance parameters). The model learns arelationship between PDF's associated with particular faces, and thelocation of those faces in the universal face space. The location in theuniversal face space of a particular face may be defined for example asthe average value of the appearance parameters representative of thatface.

During face identification using the model a PDF for an unknown face isestimated, on the basis of the position of one or more appearanceparameters representing the unknown face. This gives a face-specific PDFwhich describes how a particular face is likely to vary, which allowsmore efficient identification of the face. The face-specific PDF may bedetermined even when only a single image of the unknown face has beenseen.

An embodiment of the first aspect of the invention may be expressedmathematically as follows:

Let c be a vector of appearance model parameters.

Let p(c|x) be a PDF for the parameters, c, itself parameterised by avector of parameters, x.

For example, the PDF could be a Gaussian with covariance matrix S═S(x).

-   -   Models may include, but are not restricted to:    -   a) A model with one scaling parameter, x₁, S(x₁)=x₁S₀.    -   b) Representing the eigenvalues of the covariance matrix by x,        i.e. S(x)=P′ diag(x)P (where P is an orthogonal matrix).

Assume the model is provided with m_(i) example face images from each ofn individuals (i−1 . . . n). Let y_(i) be the mean of these m, vectorsfor individual i.

For each individual, i, the parameters x_(i) of the PDF's about themean, y₁ are found.

The model learns the relationship between the mean position in thespace, y_(i), and the parameters of the PDF, x_(i):x=f(y)

The relationship can be learnt with any suitable method, for examplemultivariate linear regression, neural networks, etc. Given sufficientdata, complex non-linear relationships can be learnt.

Thus, given a single image of an individual, the model is able to learnthe face parameters, y, and can then estimate the associated PDF asp(c|f(y)).

The PDF can be used for classification/identification in a standardmaximum likelihood classifier framework in which an object representedby parameters, c, is classified as the class j which gives the largestvalue of p_(j)(c) where p_(j)( ) is the PDF for the j'th class.

The invention may be used for example to improve recognition ofindividuals by taking into account the predicted variability in theappearance of a given individual.

The invention may also be used for example to improve tracking ofindividuals by taking into account the predicted variability in theappearance of a given individual. By providing stronger constraints onthe expected variation in appearance of an individual, matching themodel to an image or a sequence can be made more robust.

Similarly, the invention may also be applied to the synthesis of a knownface, and allows the predicted variability in the appearance of a givenindividual to be taken into account. A synthesised face may be used forexample as part of a computer game, or as part of a generaluser-computer interface. The synthesised face could be the face of theuser.

The invention also relates to using an appropriate PDF for verificationpurposes. The PDF can be used for verification by accepting an objectwith parameters c as a valid example of the class if p(c)>t₀; where t₀is a predetermined threshold. Where verification is required, the PDF ofa specific object class is first determined using a series of images ofthat object class. Once this has been done, verification may be carriedout on subsequent occasions by obtaining a single image of the objectand applying the model within the constraints of the PDF specific tothat object class.

The invention also relates to using an appropriate PDF for synthesispurposes. An image of an object is synthesised by converting appearanceparameters representative of that object in universal face space into atwo-dimensional intensity representation of the object. The appearanceparameters are confined within limits determined by the specificprobability density function of the relevant object class.

An embodiment of the fourth aspect of the invention relates to faceverification. In general, the appearance of a face can be represented bya vector of n appearance model parameters, c. However, the appearance ofan individual face can only vary in a limited number of ways, and so canbe modelled as c=c₀+Bb, where b is a k dimensional vector, k<n, B is an×k matrix, and co is the mean appearance for the individual.

When searching for a known individual (i.e. attempting to verify theidentity of an individual), it is only required to find the k parametersof b which best match the model to the image, rather than the nparameters of the full c. For faces, n≈1100 and k≈30, and the fourthaspect of the invention therefore leads to faster and more reliable facematching.

Although the invention is described in terms of a linear transformation,any suitable transformation of the form c=f(b) may be used.

The invention is useful when a series of images of a face, for example amoving face image, is to be transmitted via a telephone line. The faceof a caller is filmed by a camera, and is converted into a set ofappearance parameters. The set of appearance parameters is transformedto a set of transformed parameters in a new co-ordinate space prior totransmission. A first part of the transmission comprises thosetransformed parameters which do not vary in the new co-ordinate space,together with a description of the transformation. A second part of thetransmission comprises those transformed parameters which do vary in thenew co-ordinate space; these parameters are transmitted for each imageof the series.

The transformed parameters are transmitted to the receiver, where theyare transformed back to appearance parameters which are used tosynthesise the image.

The number of transformed parameters needed to transmit a given image tothe receiver is significantly less than the number of appearance modelparameters that would be needed to transmit the same image. This allowsa lower bandwidth connection between transmitter and receiver to beused, or alternatively allows the image to be updated more frequently.When the update rate of a face image is increased, the face image willlook more realistic, and differences between successive face images willbe less, making tracking of the face by a camera more robust.

The invention may also be applied to tracking known faces. The reductionin the number of parameters required to represent the face allows fasterand more robust tracking, which is of value in such applications asvideo-phones (in which a face tracked at one end is encoded into a smallnumber of parameters, b, which are transmitted to the receiver, wherethey are used to reconstruct a synthetic face which mimics theoriginal). The transformation required to convert the b parameters to cparameters must also be transmitted to the receiver.

The invention is of benefit when used to synthesize a known face. Thereduction in the number of parameters b required to represent the faceallows faster synthesis, which reduces the required computational load.A synthesised face may be used for example as part of a computer game,or as part of a general user-computer interface. The synthesised facecould be the face of the user.

All of the above methods the invention may be applied to the ActiveAppearance Model (G. Edwards, C. Taylor, and T. Cootes. Interpretingface images using active appearance models. In 3^(rd) InternationalConference on Automatic Face and Gesture Recognition 1998, pages300–305, Nara, Japan, April 1998. IEEE Computer Society Press) anddescribed further by Cootes et al (T. Cootes, G. J. Edwards, and C. J.Taylor. Active appearance models. In 5^(th) European Conference onComputer Vision, pages 484–498. Springer, June 1998).

The Active Appearance Model uses the difference between a reconstructedimage generated by a model and an underlying target image, to drivemodel parameters towards better values. In a prior learning stage, knowndisplacements, c, are applied to known model instances and the resultingdifference between model and image, v, is measured. Multivariate linearregression is applied to a large set of such training displacements andan approximate linear relationship is established:δc=Rδv

When searching an image, the current difference between model and image,v, is used to predict an adjustment, −c, to the model parameters whichimproves model fit. For simplicity of notation, the vector c is assumedto include displacements in scale, rotation, and translation.

The Active Appearance Model was constructed using sets of face images.To do this, Facial appearance models were generated following theapproach described by Edwards et al (G. Edwards, A. Lanitis, C Taylorand T. Cootes, Statistical model of face images—improving specificity.Image and Vision Computing, 16:203–211, 1998). The models were generatedby combining a model of face shape variation with a model of theappearance variations of a shape-normalised face. The models weretrained on 400 face images, each labelled with 122 landmark pointsrepresenting the positions of key features. The shape model wasgenerated by representing set of landmarks as a vector, x and applying aprincipal component analysis (PCA) to the data. Any example can then beapproximated using:x={overscore (x)}+P _(s) b _(s)  (1)

where {overscore (x)} is the mean shape, P_(s) is a set of orthogonalmodes of variation and b_(s) is a set of shape parameters. Each exampleimage was warped so that its control points matched the mean shape(using a triangulation algorithm), and the grey level information g wassampled from this shape-normalised face patch. By applying PCA to thisdata a similar model was obtained:g={overscore (g)}+P _(g) b _(g)  (2)

The shape and appearance of any example can thus be summarised by thevectors b_(s) and b_(g). Since there are correlations between the shapeand grey-level variations, a further PCA was applied to the concatenatedvectors, to obtain a combined model of the form:x={overscore (x)}+Q _(s) c  (3)g={overscore (g)}+Q _(g) c  (4)

where c is a vector of appearance parameters controlling both the shapeand grey-levels of the model, and Q_(s) and Q_(g) map the value of c tochanges in the shape and shape-normalised grey-level data. A face can besynthesised for a given c by generating the shape-free grey-level imagefrom the vector g and warping it using the control points described by x(this process is described in detail in G. J. Edwards, C. J. Taylor andT. Cootes, Learning to Identify and Track Faces in Image Sequences. InBritish Machine Vision Conference 1997, Colchester, UK, 1997).

The 400 examples lead to 23 shape parameters, b_(s), and 114 grey-levelparameters, b_(g). However, only 80 combined appearance modelparameters, c, are required to explain 98% of the observed variation.

Once the appearance model has been generated, it may be used to identifyfaces and to generate representations of faces.

A two-stage strategy is adopted for matching an appearance model to faceimages. The first step is to find an approximate match using a simpleand rapid approach. No initial knowledge is assumed of where the facemay lie in the image, or of its scale and orientation. A simpleeigenface model (M. Turk and A. Pentland. Eigenfaces for recognition.Journal of Cognitive Neuroscience, 3(1):71–86, 1991) may be used forthis stage of the location. A correlation score, S, between theeigenface representation of the image data, M and the image itself, Ican be calculated at various scales, positions and orientations:S=|I−M ²|  (5)

Although in principle the image could be searched exhaustively, it ismuch more efficient to use a stochastic scheme similar to that of Mataset al (K. J. J. Matas and J. Kittler. Fast face localisation andverification. In British Machine Vision Conference 1997, Colchester, UK,1997). Both the model and image are sub-sampled to calculate thecorrelation score using only a small fraction of the model samplepoints.

Once a reasonable starting approximation of the position of a face hasbeen determined, the appearance model is then used to identify the face.The parameters of the appearance model are adjusted, such that asynthetic face is generated which matches the image as closely aspossible. The basic idea is outlined below, followed by details of thealgorithm.

Interpretation is treated as an optimisation problem in which thedifference between a real face image and one synthesised by theappearance model is minimised. A difference vector δI can be defined:δI=I _(i) −I _(m)  (6)

where I_(i) is the vector of grey-level values in the image, and I_(m),is the vector of grey-level values for the current model parameters.

To locate a best match between model and image, the magnitude of thedifference vector, Δ=|δI²|, is minimised by varying the modelparameters, c.

Since the model has around 80 parameters, this appears at first to be avery difficult optimisation problem involving search in a veryhigh-dimensional space. However, it is noted that each attempt to matchthe model to a new face image, is actually a similar optimisationproblem. Therefore, the model learns something about how to solve thisclass of problems in advance. By providing a-priori knowledge of how toadjust the model parameters during image search, it arrives at anefficient run-time algorithm. In particular, it might be expected thatthe spatial pattern in δI, to encode information about how the modelparameters should be changed in order to achieve a better fit. Forexample, if the largest differences between the model and the imageoccurred at the sides of the face, that would imply that a parameterthat adjusted the width of the model face should be adjusted.

In adopting this approach there are two parts to the problem: learningthe relationship between δI and the error in the model parameters, δcand using this knowledge in an iterative algorithm for minimising Δ.

The simplest model that could be chosen for the relationship between δIand the error in the model parameters (and thus the correction whichneeds to be made) is linear:δc=RδI  (7)

This is a good enough approximation to provide good results. To find R,a multiple multivariate linear regression is performed on a large sampleof known model displacements, δc, and the corresponding differenceimages, δI. These large sets of random displacements are generated, byperturbing the ‘true’ model parameters for the images in the trainingset by a known amount. As well as perturbations in the model parameters,small displacements in 2D position, scale, and orientation are alsomodelled. These extra 4 parameters are included in the regression; forsimplicity of notation, they can, however, be regarded simply as extraelements of the vector δc. In order to obtain a well-behavedrelationship it is important to choose carefully the frame of referencein which the image difference is calculated. The most suitable frame ofreference is the shape-normalised face patch described above. Adifference is calculated thus: for the current location of the model,calculate the image grey-level sample vector, g_(i), by warping theimage data at the current location into the shape-normalised face patch.This is compared with the model grey-level sample vector, g_(m),calculated using equation 4:δg=g _(i) −g _(m)  (8)

Thus, equation 7 can be modified:δc=Rδg  (9)

The best range of values of δc to use during training is determinedexperimentally. Ideally it is desired to model a relationship that holdsover as large a range of errors, δg, as possible. However, the realrelationship is found to be linear only over a limited range of values.In experiments, the model used 80 parameters. The optimum perturbationlevel was found to be around 0.5 standard deviations (over the trainingset) for each model parameter. Each parameter was perturbed from themean by a value between 0 and 1 standard deviation. The scale, angle andposition were perturbed by values ranging from 0 to +/−10% (positionaldisplacements are relative to the face width). After performing linearregression, an R² statistic is calculated for each parameterperturbation, δc₁ to measure how well the displacement is ‘predicted’ bythe error vector δg. The average R² value for the 80 parameters was0.82, with a maximum of 0.98 (the 1st parameter) and a minimum of 0.48.

Given a method for predicting the correction which needs to be made inthe model parameters, an iterative method may be constructed for solvingthe optimisation problem. For a given model projection into the image,c, the grey-level sample error vector, δg, is calculated, and the modelestimate is updated thus:c′=c−Rδg  (10)

If the initial approximation is far from the correct solution thepredicted model parameters at the first iteration will generally not bevery accurate but should reduce the energy in the difference image. Thiscan be ensured by scaling R so that the prediction reduces the magnitudeof the difference vector, |δg²|, for all the examples in the trainingset. Given the improved value of the model parameters, the predictionmade in the next iteration should be better. The procedure is iteratedto convergence. Typically the algorithm converges in around 5–10iterations from fairly poor starting approximations.

The invention may be applied to the Active Appearance Model. In thegeneral form, given appearance parameters, c, a synthetic image isgenerated, and a difference vector, I, between the synthetic image andthe target image is computed. The appearance parameters are then updatedusing the equation:c→c−RδI

where R is a regression matrix relating the model displacement to theimage errors, learnt during the model training phase.

Applying the invention, when the model is looking for a particularindividual with known mean appearance co and variation described by B,then the parameters b can be manipulated as follows:b→b−R _(B) δIc=c _(o) +Bb

where R_(B) is a regression matrix learnt from the training set in a wayanalogous to that used for computing R, but learning the relationshipbetween small changes in b and the induced image error.

When verifying that an image is of a particular object, it is assumedthat the mean appearance, c_(o), and the way in which the object maylegitimately vary, B, (c=c_(o)+Bb) are known The image is searched usingan active appearance model manipulating the reduced set of parameters,b. The best fit will synthesise an image of the target object as closeas possible to the target image. To verify that the object belongs tothe required class, the difference between the best fitting synthesisedimage and the actual image, dI, is measured. If |dI|<t₁, where t₁ is asuitable threshold, the object is declared to be correctly verified.

The algorithm used in the prior art Active Appearance Model uses thesame regression matrix, R, for all individuals. The seventh aspect ofthe invention uses a different R for each individual (i.e. for eachsubject's face). Specifically, if R is represented within a model with tparameters, x, i.e. R R(x), then the relationship between the parametersx and the mean appearance model parameters can be learnt for anindividual, y, i.e.x=g(y).

Thus, when searching for an individual with mean appearance modelparameters, y, the model uses an update equation of the form:c→c−R(g(y))δI

For example, consider one simple model having a single parameter, x,i.e.R(x)=xR ₀.

It may be that there is a relationship between distance from the originand the best value of x, i.e. x=a+b|y|.

The regression matrix can be calculated at any point in the space, y, byusing the model to synthesise an image for the parameters y, thengenerating large numbers of displacements, dy, and corresponding imageerrors δI. Linear regression can be used to learn the best R such thatdy=RδI for the given y.

1. A method of identifying an object class using a model based uponappearance parameters derived by comparing images of objects ofdifferent classes, the model including a representation of a probabilitydensity function describing a range over which appearance parameters mayvary for a given class of object, the model further including a definedrelationship between the shape of the probability density function andthe location of the probability density function in appearance space,wherein the method comprises: generating appearance parametersrepresentative of an unknown object, estimating an appropriateprobability density function for the unknown object based upon thelocation of the appearance parameters in appearance space and using thedefined relationship between the shape of the probability densityfunction and the location of the probability density function inappearance space, then iteratively modifying at least some of theappearance parameters within limits determined using the probabilitydensity function to provide a set of appearance parameters whichidentify an object class.
 2. A method according to claim 1, wherein athreshold level of the probability density function is determined, andthe appearance parameters are constrained to have a probability densitygreater than the threshold.
 3. A method according to claim 1, whereinthe objects are faces, and a given class of object is a face having aparticular identity.
 4. A method according to claim 1, wherein therelationship between the appearance parameters and the probabilitydensity function is a relationship between average appearance parametersdetermined for each class of object, and probability density functionsassociated with each class of object.
 5. A method according to claim 1,wherein the probability density function is a Gaussian function.
 6. Amethod according to claim 1, wherein the probability density function isapproximated as a Gaussian with a given covariance matrix.
 7. A methodaccording to claim 1, wherein the model is the Active Appearance Model.8. A method of verifying the identity of an object class using a modelbased upon appearance parameters derived by comparing images of objectsof different classes, the model including a representation of aprobability density function describing the variation of appearanceparameters for a given class of object, the model function including adefined relationship between the shape of the probability densityfunction and the location of the probability density function inappearance space, wherein for an object class for which verificationwill subsequently be required, the method comprises: using a series ofimages of that object class to generate a representation of a specificprobability density function describing the variation of appearanceparameters for that object class, and during subsequent verification ofthe object class, comparing an image of an object of an unknown classwith the object of known class by generating appearance parametersrepresentative of the object of unknown class and iteratively modifyingat least some of the appearance parameters within limits determinedusing the specific probability density function, to provide a set ofappearance parameters which are compared with a predetermined thresholdto provide verification.
 9. A method according to claim 8, wherein athreshold level of the probability density function is determined, andthe appearance parameters are constrained to have a probability densitygreater than the threshold.
 10. A method according to claim 8, whereinthe objects are faces, and a given class of object is a face having aparticular identity.
 11. A method according to claim 8, wherein therelationship between the appearance parameters and the probabilitydensity function is a relationship between average appearance parametersdetermined for each class of object, and probability density functionsassociated with each class of object.
 12. A method according to claim 8,wherein the probability density function is a Gaussian function.
 13. Amethod according to claim 8, wherein the probability density function isapproximated as a Gaussian with a given covariance matrix.
 14. A methodaccording to claim 8, wherein the model is the Active Appearance Model.15. A method of generating a synthesized image of an object class usinga model based upon appearance parameters derived by comparing images ofobjects of different classes, the model including a representation of aprobability density function describing the variation of appearanceparameters for a given class of object, the model further including adefined relationship between the shape of the probability densityfunction and the location of the probability density function inappearance space, wherein for an object class to be synthesized, themethod comprises: using a series of images of that object class togenerate a representation of a specific probability density functiondescribing the variation of appearance parameters for that object class,and during subsequent synthesis of the object class, confining theappearance parameters used within limits determined using the specificprobability density function.
 16. A method according to claim 15,wherein a threshold level of the probability density function isdetermined, and the appearance parameters are constrained to have aprobability density greater than the threshold.
 17. A method accordingto claim 15, wherein the objects are faces, and a given class of objectis a face having a particular identity.
 18. A method according to claim15, wherein the relationship between the appearance parameters and theprobability density function is a relationship between averageappearance parameters determined for each class of object, andprobability density functions associated with each class of object. 19.A method according to claim 15, wherein the probability density functionis a Gaussian function.
 20. A method according to claim 15, wherein theprobability density function is approximated as a Gaussian with a givencovariance matrix.
 21. A method according to claim 15, wherein the modelis the Active Appearance Model.